Image capture device calibration

ABSTRACT

A three-dimensional coordinate position of a calibration device is determined. Further, a code is emitted to an image capture device. The code indicates the three-dimensional coordinate position of the calibration device. In addition, an image of light emitted from the calibration device is captured. The light includes the code. An image capture device three-dimensional coordinate position of the calibration device is calibrated according to the real world three-dimensional coordinate position of the calibration device indicated by the code.

BACKGROUND

1. Field

This disclosure generally relates to the field of image capture systems.More particularly, the disclosure relates to camera calibration.

2. General Background

Camera calibration is currently utilized in a variety of areas to findthe correspondence of a real world coordinate system with a coordinatesystem of a camera. Camera calibration is often utilized to calibrateprojector-camera systems. Further, camera calibration is often utilizedto calibrate multi-camera systems that track people and theirinteractions.

Current approaches to calibrating one or more cameras utilize acheckerboard target, a distinctive point such as a light emitting diode(“LED”) light source that is moved from position to position, ormultiple naturally occurring points in a scene. In the approach thatutilizes the checkerboard target, the one or more cameras are typicallydirected toward the checkerboard target that is typically held by ahuman. The calibration involves the human adjusting the checkerboard tobe located at different ranges from the camera with differentinclinations. In the approach that utilizes a moving distinctive point,the one or more cameras are typically directed toward the distinctivepoint that is typically held by a human. The calibration involves thehuman moving the distinctive point through a variety of positions. Withrespect to the approach that utilizes multiple naturally occurringpoints in a scene, the multiple naturally occurring points must have anappropriate distribution in the scene. Otherwise, the calibration willnot be correct, e.g., the calibration will not be correct if thenaturally occurring points lie on a line or a plane.

The approaches with the checkerboard and the moving distinctive pointoften have unskilled workers performing the tasks incorrectly.Uncertainty and wasted time often result. If the human does not performthe calibration task correctly, the calibration approaches do notprovide helpful diagnostic messages that aid the user in performing thecalibration task correctly.

Further, the current calibration approaches do not fully specify thereal world coordinate system because they do not specify the position ofthe origin of the real world coordinate system or the orientation of theaxes of the real world coordinate system. An additional step isnecessary to fully specify the real world coordinate system, whichtypically involves a human providing a manual input for the cameraimages. This additional step requires extra work and is often a sourceof error.

In addition, the current calibration approaches do not specify sceneinformation such as the location of the ground plane in the worldcoordinate frame. An additional step is necessary to specify suchinformation, which typically involves a human providing a manual inputfor the camera images. This additional step also requires extra work andis often a source of error.

It is believed that improvements are needed to provide betterdiagnostics for camera calibration and to provide a better methodologyto specify the origin of the world coordinate frame and the orientationof the axes. It is also believed that improvements are needed to providea better methodology to specify scene information such as the locationof the ground plane.

SUMMARY

In one aspect of the disclosure, a calibration device is described. Thecalibration device comprises a controller that determines athree-dimensional coordinate position of the calibration device.Further, the calibration device comprises at least one light emitterthat emits a code to an image capture device. The code indicates thethree-dimensional coordinate position of the calibration device.

In another aspect of the disclosure, an image capture device isdescribed. The image capture device comprises an image capture sensorthat captures an image of light emitted from a calibration device. Thelight includes a code that indicates a real world three-dimensionalcoordinate position of the calibration device. Further, the imagecapture device comprises a controller that calibrates at least oneintrinsic parameter and at least one extrinsic parameter according to areal world three-dimensional coordinate position of the calibrationdevice indicated by the code.

In yet another aspect of the disclosure, a system is described. Thesystem comprises a calibration device that includes a controller and atleast one light emitter. The controller determines a three-dimensionalcoordinate position of the calibration device, the at least one lightemitter emitting a code to an image capture device. The code indicatesthe three-dimensional coordinate position of the calibration device.Further, the system comprises an image capture device that includes animage capture sensor and a controller. The image capture sensor capturesan image of light emitted from a calibration device, the light includingthe code. The controller calibrates an image capture devicethree-dimensional coordinate position of the calibration deviceaccording to the real world three-dimensional coordinate position of thecalibration device indicated by the code.

In another aspect of the disclosure, a process is described. The processdetermines a set of three-dimensional coordinate positions of acalibration device. Further, the process emits a code to an imagecapture device. The code indicates the three-dimensional coordinateposition of the calibration device. In addition, the process captures animage of light emitted from the calibration device. The light includesthe code. The process also calibrates an image capture devicethree-dimensional coordinate position of the calibration deviceaccording to the set of real world three-dimensional coordinatepositions of the calibration device indicated by the code.

In yet another aspect, an image capture device is described. The imagecapture device comprises a lens. Further, the image capture devicecomprises an image capture sensor that captures an image of lightemitted from a calibration device through the lens, the light includinga code that indicates a real world three-dimensional coordinate positionof the calibration device.

In another aspect, a robotic calibration apparatus is described. Therobotic calibration apparatus comprises a calibration device thatincludes a controller and at least one light emitter. The controllerdetermines a three-dimensional coordinate position of the calibrationdevice, The at least one light emitter emits a code to an image capturedevice. The code indicates the three-dimensional coordinate position ofthe calibration device. Further, the robotic calibration apparatuscomprises a robotic platform that is operably connected to thecalibration device to move the at least one light emitter to anadditional three-dimensional coordinate position that is distinct fromthe three-dimensional coordinate position.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned features of the present disclosure will become moreapparent with reference to the following description taken inconjunction with the accompanying drawings wherein like referencenumerals denote like elements and in which:

FIG. 1 illustrates a calibration system.

FIG. 2 illustrates a calibration configuration in which a coded patternis blinked for calibration.

FIGS. 3A and 3B illustrate variations in which image data and locationdata may be received by different hardware.

FIG. 3A illustrates a calibration configuration in which an opticalreceiver distinct from the image capture device receives the locationdata from the calibration device.

FIG. 3B illustrates a calibration configuration in which the opticalreceiver is connected directly to the image capture device.

FIG. 4 illustrates a robotic configuration that is utilized forcalibration.

FIG. 5 illustrates a process that is utilized to provide calibration.

DETAILED DESCRIPTION

A calibration system is utilized to calibrate a coordinate system of animage capture device with a real world coordinate system. Thecalibration system improves image capture calibration by automaticallyproviding diagnostics in the event that calibration was not performedcorrectly. Such information may be helpful to a user that is involved inthe calibration process. Further, particular directions are provided tothe user on actions to be performed to obtain a correct calibration.

FIG. 1 illustrates a calibration system 100. The camera calibrationsystem includes an image capture device 102, a calibration device 110,and a transmitter 122. An example of an image capture device 102 is acamera. The image capture device 102 has a controller 104, a lens 106,and an image capture device three-dimensional coordinate system 108. Theimage capture device three-dimensional coordinate system 108 is anexample of intrinsic data for the lens 106 of the image capture device102. In one aspect, transmitter 122 is utilized to provide diagnosticdata in the event of incorrect calibration. The transmitter is operablyconnected to the calibration device 112, e.g., through a cableconnection, a wireless connection, or the like. In another aspect, thecalibration system 100 does not utilize the transmitter 122, e.g.,diagnostic data may be provided from human interaction such as verbal orvisual cues. A user analyzes a set of (x, y, z) positions and determinesif the calibration device 110 was properly moved around the workspace.That user provides instructions to another user operating thecalibration device 110 to move to new positions in the workspace.

Further, the calibration device 110 has a light emitter 112, an inputdevice 114, and a controller 116. The light emitter 112 emits light suchas visible light, infrared light, RGB light, or the like. Further, thelight emitter 112 may emit a combination of various types of light. Thelight emitter 112 is illustrated as being attached to the exterior ofthe calibration device 112. As an example, a light emitting diode(“LED”) or a plurality of LEDs is utilized as the light emitter 112.Alternatively, the light emitter 112 is integrated within thecalibration device 110. As another alternative, the light emitter 112 isa separate device that is operably connected to the calibration device110 through a cable connection, wireless connection, or the like.

In one aspect, a user utilizes the input device 114 to input data. Anexample of the input device 114 is a keypad with letters and/or numbers.A user enters data relevant to the position of the calibration device110 into the input device 114. As an example, the user entersinformation about particular corners of a location that is beingcalibrated. In another aspect, the calibration device 112 does not havethe input device 114 as user input may not be necessary.

In one aspect, the calibration device 112 has an output device 116. Anexample of the output device 116 is a display screen. The output device116 is utilized to display diagnostic data received from the transmitter122. In another aspect, the calibration device 112 does not utilize theoutput device 116. In such an instance, the calibration device 112 mayreceive diagnostic data through human interaction.

The controller 118 determines the location of the calibration device110, i.e., the real world three-dimensional coordinate position on areal world three dimensional coordinate system 120. The controller 118may utilize one or more of a variety of different location systems todetermine the location of the calibration device 110, e.g., GPS systemsor the like.

The calibration device 110 acts as an intelligent calibration target forthe image capture device 102. The controller 118 provides the x, y, andz coordinates of the real world position of the calibration device 110to the light emitter 112 so that the light emitter 112 visually emits orvisually broadcasts a code with the current (x, y, z) position. Theimage capture device 102 or any other image capture device that viewsthe light emitter 112 can decode and record the current position of thelight emitter 112 along with the corresponding image position. The imagecapture device 102 or an operably connected processor performscalibration from the real world three dimensional coordinate system 118to the image capture device three-dimensional coordinate system 108. Thecode may include a single (x, y, z) position. As a result, multiplecodes may be emitted such that each code represents the current (x, y,z) position. Alternatively, a code may include multiple (x, y, z)positions, e.g., each code is sequenced according to time.

In one aspect, the camera calibration device 110 has or is in operablecommunication with an encoder that encodes the three-dimensionalcoordinate position of the calibration device 110 into the code.Further, the camera calibration device 110 has or is in operablecommunication with a modulator that modulates the code into a lightsignal emitted by the at least one light emitter 112. The positioninformation is modulated into the light signal utilizing one or moremodulation techniques, e.g., amplitude modulation, frequency modulation,pulse-width modulation (“PWM”), or the like.

In another aspect, the image capture device 102 has or is in operablecommunication with a demodulator that demodulates the light from thecalibration device to obtain the code. Further, the image capture device102 has or is in operable communication with a decoder that decodes thecode to determine the set of real world three-dimensional coordinatepositions of the calibration device.

This methodology determines a set of correspondences of (x. y, z)positions of the moving calibration target and the corresponding (x, y)pixels of the image plane. Such data provides enough information, e.g.,intrinsics, extrinsics, and radial distortion, for calibration of theimage capture device 102. The controller 104 performs the calibration.Alternatively, a processor of another device that is operably connectedto or in operable communication with the image capture device 102performs the calibration.

The controller 104 is illustrated as being integrated within the imagecapture device 102. Alternatively, the controller 104 may be distinctfrom the image capture device 102 and operably connected to the imagecapture device 102.

The calibration device 110 may be implemented in a variety ofconfigurations. As an example, the calibration device 110 is a mobiledevice that a user carries from position to position so that the imagecapture device 102 captures the light emitted from the light emitter112. The mobile device may also be an existing computing device, e.g., anotebook computer, a laptop, a smartphone, a tablet device, or the like,that is adapted for utilization as the calibration device 110. As anexample, a tablet device is utilized as the calibration device 110. Thedisplay screen of the tablet device is utilized as the light emitter 112to display a coded pattern with the (x, y, z) position of thecalibration device 110 that is captured by the image capture device 102.As another example, the calibration device 110 is implemented on arobotic platform that automatically moves the light emitter 112 througha sequence of positions. The robotic platform may be placed on theground by the user or may be positioned on a small mobile platform thatis capable of traversing the workspace. Utilizing the robotic platformremoves human error from the calibration procedure.

Further, the calibration device 110 may include onboard 3D sensors toobtain 3D information about the scene. An example of an onboard 3Dsensor that is utilized to obtain 3D information about a scene is analtimeter that senses the height of the calibration device 110 aboveground level. Other examples include a laser tape measure, a laserscanner, a depth sensor that detects 3D data for the surroundings, aGlobal Positioning System (“GPS”), a Wi-Fi positioning system, aninfrared (“IR”) positioning system, or the like. The onboard 3D sensorsare utilized for calibration with scene information in contrast withprevious approaches that calibrate cameras without scene information,i.e., by separately generating a 3D model of the scene and thenregistering the coordinate frame of the calibrated cameras with thecoordinate frame of the scene model. The onboard 3D sensors allow for ascene model to be generated in conjunction with the calibration of thecameras.

In one aspect, the code includes both the current (x, y, z) position andthe scene information. In another aspect, different codes are utilizedfor the current (x, y, z) position and the scene information.

FIG. 2 illustrates a calibration configuration 200 in which a codedpattern is blinked for a calibration. The image capture device 102 isillustrated as a camera for illustrative purposes in FIG. 2. The lightemitter 112 blinks the location information of the calibration device110 in a coded pattern. The data and the visual scene information arecaptured by the image capture device 102 through a camera lens. In suchan instance, the camera frame rate is utilized to measure the potentialspeed of the light emission by the light emitter 112. For example, acamera speed of thirty frames per second allows for one blink by thelight emitter 112 every one thirtieth of a second.

Although a single LED is illustrated, a plurality of LEDs may beutilized as the light emitter 112. As an example, twenty four LEDs eachsimultaneously blink a particular bit of information. As a result,twenty four bits of information are emitted simultaneously. Eachcoordinate is represented by eight bits such that an (x, y, z)coordinate is represented by the twenty four bits of information, i.e.,eight bits for the x coordinate, eight bits for the y coordinate, andeight bits for the z coordinate. As yet another alternative, thevertices of a cube each emit a bit of information that compose to formthe code with the (x, y, z) coordinate. As a result of utilizing aplurality of LEDs, the (x, y, z) coordinate information may be emittedrelatively fast.

The calibration configuration 200 illustrated in FIG. 2 utilizes animage capture device 102 that functions to receive the location datafrom the calibration device 110 in addition to the image data that animage capture device 102 typically receives. FIGS. 3A and 3B illustratevariations in which image data and location data may be received bydifferent hardware.

FIG. 3A illustrates a calibration configuration 300 in which an opticalreceiver 302 distinct from the image capture device 102 receives thelocation data from the calibration device 110. The optical receiver 302receives the location data at a faster rather than that of the imagecapture device 110. As an example, an infrared receiver is operablyconnected to a controller 304, which is operably connected to the imagecapture device 102. The operably connections may be cable connections,wireless connections, or the like. The light emitter 112 emits light ata high rate such that the light appears solid to the image capturedevice 102, but is perceivable as blinked light by the optical receiver302.

The controller 304 is illustrated as being distinct from the imagecapture device 102. In one aspect, the controller 104 performs thefunctionality described with respect to FIG. 1 and the controller 304coordinates the interaction between the image capture device 102 and theoptical receiver 302. In another aspect, the controller 104 performs thefunctionality described with respect to FIG. 1 and the coordinationbetween the image capture device 102 and the optical receiver 302. Inyet another aspect, the image capture device 102 does not have thecontroller 104 and relies on the controller 304 to perform thecontroller functionality described with respect to FIG. 1 and thecoordination between the image capture device 102 and the opticalreceiver 302.

FIG. 3B illustrates a calibration configuration 350 in which the opticalreceiver 302 is connected directly to the image capture device 302. Asan example, the optical receiver 302 is a photocell or other opticaldetector that is connected directly to the image capture device 102.

FIG. 4 illustrates a robotic configuration 400 that is utilized forcalibration. The light emitter 112 is positioned on a robotic platform402. The robotic platform 402 may move the light emitter 112 along thex, y, and/or z coordinate planes. For example, the robotic platform 402moves a rod 404 up and down along the y coordinate plane. Further, therobotic platform 402 moves a base plate 406 forward or backward along anx coordinate plane and/or sideways along a z coordinate plane. In oneaspect, the robotic platform 402 has a plurality of wheels 408 or othermovement apparatus that moves the robotic platform 402 to differentlocations. Movement of the rod 404 and the base plate 406 may beutilized in conjunction with or independently of the plurality of wheels408.

In one aspect, the robotic platform 402 is instructed to automaticallymove the light emitter 112 to various positions. As a result, lightemitter is able to emit absolute (x, y, z) coordinates without humanerror to the image capture device 102. Further, the robotic platform 402assists in identifying the ground plane as the robotic platform isresting on the ground.

FIG. 5 illustrates a process 500 that is utilized to providecalibration. At a process block 502, the process 500 process determinesa three-dimensional coordinate position of a calibration device.Further, at a process block 504, the process 500 emits a code to animage capture device. The code indicates the three-dimensionalcoordinate position of the calibration device. In addition, at a processblock 506, the process 500 captures an image of light emitted from thecalibration device. The light includes the code. At a process block 508,the process 500 also calibrates an image capture devicethree-dimensional coordinate position of the calibration deviceaccording to the real world three-dimensional coordinate position of thecalibration device indicated by the code.

In another aspect, the light emitter 112 illustrated in FIG. 1 may havemetric properties. Such metric properties allow for faster and morereliable measurements. As an example, a plurality of LEDs is spacedapart at predetermined distances along a rod. Such measurements allowfor reliable identification of the ground plane. Further, othermeasurement information may be utilized such as only allowing a rod toemit light if the rod is held absolutely vertical.

In yet another aspect, the calibration methodology is utilized for anexpanding structure. A plurality of LEDs is mounted at particularlocations of the expanding structure. As the structure expands, aplurality of vectors is measured.

The processes described herein may be implemented in a general,multi-purpose or single purpose processor. Such a processor will executeinstructions, either at the assembly, compiled or machine-level, toperform the processes. Those instructions can be written by one ofordinary skill in the art following the description of the figurescorresponding to the processes and stored or transmitted on a computerreadable medium. The instructions may also be created using source codeor a computer-aided design tool. A computer readable medium may be anymedium capable of carrying those instructions and include a CD-ROM, DVD,magnetic or other optical disc, tape, silicon memory (e.g., removable,non-removable, volatile or non-volatile), packetized or non-packetizeddata through wireline or wireless transmissions locally or remotelythrough a network. A computer is herein intended to include any devicethat has a general, multi-purpose or single purpose processor asdescribed above. For example, a computer may be a personal computer(“PC”), laptop, smartphone, tablet device, set top box, or the like.

It is understood that the apparatuses, systems, computer programproducts, and processes described herein may also be applied in othertypes of apparatuses, systems, computer program products, and processes.Those skilled in the art will appreciate that the various adaptationsand modifications of the aspects of the apparatuses, systems, computerprogram products, and processes described herein may be configuredwithout departing from the scope and spirit of the present apparatuses,systems, computer program products, and processes. Therefore, it is tobe understood that, within the scope of the appended claims, the presentapparatuses, systems, computer program products, and processes may bepracticed other than as specifically described herein.

We claim:
 1. An image capture device comprising: an image capture sensor that captures an image of light emitted from a calibration device, the light including a code that indicates a set of real world three-dimensional coordinate positions of the calibration device; and a controller that calibrates at least one intrinsic parameter and at least one extrinsic parameter according to the set of real world three-dimensional coordinate positions of the calibration device indicated by the code.
 2. The image capture device of claim 1, further comprising a transmitter that sends diagnostic data to the calibration device based upon the calibration.
 3. The image capture device of claim 1, wherein the controller decodes the code to determine the real world three-dimensional coordinate position of the calibration device.
 4. The camera calibration device of claim 1, further comprising a demodulator that demodulates the light from the calibration device to obtain the code.
 5. The camera calibration device of claim 4, further comprising a decoder that decodes the code to determine the set of real world three-dimensional coordinate positions of the calibration device.
 6. A system comprising: a calibration device that includes a controller and at least one light emitter, the controller determining a three-dimensional coordinate position of the calibration device, the at least one light emitter emitting a code to an image capture device, the code indicating the three-dimensional coordinate position of the calibration device; and an image capture device that includes an image capture sensor and a controller, the image capture sensor capturing an image of light emitted from a calibration device, the light including the code, the controller calibrating an image capture device three-dimensional coordinate position of the calibration device according to the real world three-dimensional coordinate position of the calibration device indicated by the code.
 7. The system of claim 6, wherein the at least one light emitter is at least one LED emitter.
 8. The system of claim 6, wherein the at least one light emitter is a display screen of the calibration device.
 9. The system of claim 6, wherein the calibration device further includes a receiver that receives diagnostic data from the image capture device after the image capture device performs calibration based upon the code.
 10. The system of claim 9, wherein the three-dimensional coordinate position is adjusted based upon the diagnostic data.
 11. The system of claim 6, wherein the calibration device further includes an input receiver that receives an input indicating data in addition to the three-dimensional coordinate position.
 12. The system of claim 11, wherein the code includes the input in addition to the three-dimensional coordinate position.
 13. The system of claim 6, wherein the image capture device further includes a transmitter that sends diagnostic data to the calibration device based upon the calibration.
 14. The system of claim 6, wherein the controller decodes the code to determine the real world three-dimensional coordinate position of the calibration device.
 15. A method comprising: determining a three-dimensional coordinate position of a calibration device; emitting a code to an image capture device, the code indicating the three-dimensional coordinate position of the calibration device; capturing an image of light emitted from the calibration device, the light including the code; and calibrating an image capture device three-dimensional coordinate position of the calibration device according to the real world three-dimensional coordinate position of the calibration device indicated by the code.
 16. The method of claim 15, wherein the at least one light emitter is at least one LED emitter.
 17. The method of claim 15, wherein the at least one light emitter is a display screen of the calibration device.
 18. The method of claim 15, wherein the calibration device further includes a receiver that receives diagnostic data from the image capture device after the image capture device performs calibration based upon the code.
 19. The method of claim 18, further comprising adjusting the three-dimensional coordinate position based upon the diagnostic data.
 20. The method of claim 15, wherein the calibration device further includes an input receiver that receives an input indicating data in addition to the three-dimensional coordinate position.
 21. The method of claim 20, wherein the code includes the input in addition to the three-dimensional coordinate position.
 22. The method of claim 15, further comprising sending diagnostic data to the calibration device based upon the calibration.
 23. The method of claim 15, further comprising decoding the code to determine the real world three-dimensional coordinate position of the calibration device. 